{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COVID-19 Deep Learning\n",
    "\n",
    "## Dataset\n",
    "\n",
    "The file covid19.csv contains daily new cases for a selection of countries thru April 1, which was scrapped from https://www.worldometers.info/coronavirus/.\n",
    "\n",
    "A more comprehensive dataset from John Hopkins University can be obtained as a Google Cloud hosted public dataset using BigQuery.\n",
    "\n",
    "## Helper Functions\n",
    "\n",
    "*blankImage(size)*\n",
    "\n",
    "Creates an blank (0's) grayscale image of shape (size, size)\n",
    "\n",
    "*getData(csv_file)*\n",
    "\n",
    "Reads in the tabular data from the specified CSV file into a pandas dataframe. The columns are:\n",
    "\n",
    "ISO 3166A2 Country Code (e.g., US for United States)\n",
    "Date (US Standard Format mm/dd/yyyy)\n",
    "New Cases\n",
    "\n",
    "*preData(pd)*\n",
    "\n",
    "Extracts the data from the panda dataframe into list format which can be used to generate bar graph images.<br/>\n",
    "\n",
    "\\[ country code, [ list of daily counts ] ]\n",
    "\n",
    "*genImages(all_data)*\n",
    "\n",
    "Uses the data from *prepData* to generate bar graph images, as follows:\n",
    "\n",
    "    1. The bar graphs are placed under the folder covid-images.\n",
    "    2. Bar graph images  are generated for each country and are prefixed with the country code.\n",
    "    3. For each country, the range of new cases is normalized between 0 and 1.\n",
    "    3. A date sequence (progression) of bar graphs is generated for each date and the sequence order is appended to the image filename.\n",
    "    4. For each bar graph image, the normalized new cases value for the next subsequent data is appended to the image filename.\n",
    "    \n",
    "Ex.  US-1-0.017.jpg\n",
    "\n",
    "US, Sequence Count 1, normalized next day new cases 0.017\n",
    "\n",
    "*preprocessData()*\n",
    "\n",
    "Reads the bar graph images into memory as a training dataset:\n",
    "\n",
    "    1. The normalized next day new cases in filename is used as the label.\n",
    "    2. Image pixel data is normalized.\n",
    "    3. The training dataset is randomly shuffled.\n",
    "    \n",
    "*prepInput(path)*\n",
    "\n",
    "Prepares a bar graph image for input to the model for prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import shutil\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "def blankImage(size):\n",
    "    ''' Create a blank grayscale image as size x size '''\n",
    "    image = np.zeros([size, size])\n",
    "    return image\n",
    "\n",
    "def getData(csv):\n",
    "    ''' Get the COVID-19 data '''\n",
    "    data = pd.read_csv(csv)\n",
    "    return data\n",
    "\n",
    "def prepData(pd, source='worldometers'):\n",
    "    ''' Prep the COVID-19 panda data into form for generating images '''\n",
    "        \n",
    "    # Per Country Data (normalized new case counts in date sequential order)\n",
    "    data_by_cc = []\n",
    "\n",
    "    c_cc   = None # current country code\n",
    "    c_data = []   # new case list for current country\n",
    "    for index, row in pd.iterrows():\n",
    "        if source == 'worldometers':\n",
    "            # extract feature values\n",
    "            cc = row['ISO 3166 A2']\n",
    "            date = row['Date']\n",
    "            cases = row['New Cases']\n",
    "        elif source == 'ecdc':\n",
    "            try:\n",
    "                if 'JPG' in row['geoId']:\n",
    "                    continue\n",
    "            except:\n",
    "                continue\n",
    "            if row['year'] == 1999:\n",
    "                continue\n",
    "            if ['geoId'] != 'CN' and row['month'] == 1:\n",
    "                continue\n",
    "            if ['geoId'] != 'CN' and row['month'] == 2 and row['day'] < 15:\n",
    "                continue\n",
    "            cc = row['geoId']\n",
    "            date = row['dateRep']\n",
    "            cases = row['cases']\n",
    "        \n",
    "        # special case, 0's are a problem, change to a 1\n",
    "        if cases == 0:\n",
    "            cases = 1\n",
    "            \n",
    "        # special case, CH one day correction\n",
    "        if cc == 'CN' and cases > 5000:\n",
    "            cases = 2500\n",
    "            print(\"FIXED CN\")\n",
    "            \n",
    "        # start new country\n",
    "        if c_cc != cc:\n",
    "            # complete preparation of previous country\n",
    "            if c_cc != None:\n",
    "                 # Reverse the row order if source is ECDC (https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-distribution-covid-19-cases-worldwide)\n",
    "                if source == 'ecdc':\n",
    "                    c_data = np.flip(c_data)\n",
    "                # normalize the data\n",
    "                max = np.max(c_data)\n",
    "                c_data = np.asarray(c_data) / max\n",
    "                data_by_cc.append((c_cc, c_data, max))\n",
    "            c_cc = cc\n",
    "            c_data = []\n",
    "        c_data.append(cases)\n",
    "        \n",
    "    return data_by_cc\n",
    "\n",
    "def genImages(all_data, size=224, bar=2):\n",
    "    ''' Generate Images for Training Data '''\n",
    "    try:\n",
    "        shutil.rmtree(\"covid-images\")\n",
    "    except:\n",
    "        pass\n",
    "    os.mkdir(\"covid-images\")\n",
    "    for cc_data in all_data:\n",
    "        cc = cc_data[0]\n",
    "        x_data = cc_data[1]\n",
    "\n",
    "        image = blankImage(size)\n",
    "        for index, cases in enumerate(x_data):\n",
    "            if cases < 2e-3:\n",
    "                cases = 1e-2\n",
    "            r = int(size * cases)\n",
    "            c = index * (bar * 2)\n",
    "            image[-r:,c:c+bar] = 255\n",
    "            if index < len(x_data) - 1:\n",
    "                next = x_data[index+1]\n",
    "                if next < 2e-3:\n",
    "                    next = 1e-2\n",
    "                cv2.imwrite(\"covid-images/\" + cc + \"-\" + str(index) + \"-\" + str(next) + \".jpg\", image)\n",
    "            # special case: last date won't have next day's total\n",
    "            else:\n",
    "                cv2.imwrite(\"covid-images/\" + cc + \"- last.jpg\", image)\n",
    "                \n",
    "def preprocessData():\n",
    "    ''' Read the Bar Graph Images into memory as a training dataset '''\n",
    "    x_data = []\n",
    "    y_data = [] # the next day count will be the label\n",
    "    files = os.scandir('covid-images')\n",
    "    for file in files:\n",
    "        # don't use the last date in training data\n",
    "        if \"last\" in file.name:\n",
    "            continue\n",
    "        next_day = file.name.split('-')[2][:-4]\n",
    "        image = cv2.imread(file.path, cv2.IMREAD_GRAYSCALE)\n",
    "        x_data.append(image)\n",
    "        y_data.append(next_day)\n",
    "        \n",
    "    # normalize the image pixel data\n",
    "    x_data = (np.asarray(x_data) / 255.0).astype(np.float32)\n",
    "    y_data = np.asarray(y_data).astype(np.float32)\n",
    "    \n",
    "    # randomly shuffle the data\n",
    "    return train_test_split(x_data, y_data, test_size=0.05, shuffle=True)\n",
    "                \n",
    "def preprocess_input(path):\n",
    "    image = cv2.imread(path, cv2.IMREAD_GRAYSCALE)\n",
    "    image = (image / 255.0).astype(np.float32)\n",
    "    return image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the Dataset for Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FIXED CN\n",
      "FIXED CN\n"
     ]
    }
   ],
   "source": [
    "# Prepare the training data\n",
    "data = getData(\"covid19.csv\")\n",
    "all_data = prepData(data, source='worldometers')\n",
    "genImages(all_data)\n",
    "x_train, x_test, y_train, y_test = preprocessData()\n",
    "# Add a single channel, ie (224, 224) -> (224, 224, 1)\n",
    "x_train = np.expand_dims(x_train, -1)\n",
    "x_test  = np.expand_dims(x_test, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the ResNet V2 model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 224, 224, 1) 0                                            \n",
      "__________________________________________________________________________________________________\n",
      "zero_padding2d (ZeroPadding2D)  (None, 230, 230, 1)  0           input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv2d (Conv2D)                 (None, 112, 112, 64) 3136        zero_padding2d[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization (BatchNorma (None, 112, 112, 64) 256         conv2d[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu (ReLU)                    (None, 112, 112, 64) 0           batch_normalization[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "zero_padding2d_1 (ZeroPadding2D (None, 114, 114, 64) 0           re_lu[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D)    (None, 56, 56, 64)   0           zero_padding2d_1[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 56, 56, 64)   256         max_pooling2d[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_1 (ReLU)                  (None, 56, 56, 64)   0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_2 (Conv2D)               (None, 56, 56, 64)   4096        re_lu_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 56, 56, 64)   256         conv2d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_2 (ReLU)                  (None, 56, 56, 64)   0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_3 (Conv2D)               (None, 56, 56, 64)   36864       re_lu_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 56, 56, 64)   256         conv2d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_3 (ReLU)                  (None, 56, 56, 64)   0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 56, 56, 64)   256         max_pooling2d[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_4 (Conv2D)               (None, 56, 56, 256)  16384       re_lu_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_1 (Conv2D)               (None, 56, 56, 256)  16384       batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add (Add)                       (None, 56, 56, 256)  0           conv2d_4[0][0]                   \n",
      "                                                                 conv2d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 56, 56, 256)  1024        add[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_4 (ReLU)                  (None, 56, 56, 256)  0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_5 (Conv2D)               (None, 56, 56, 64)   16384       re_lu_4[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, 56, 56, 64)   256         conv2d_5[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_5 (ReLU)                  (None, 56, 56, 64)   0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_6 (Conv2D)               (None, 56, 56, 64)   36864       re_lu_5[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_7 (BatchNor (None, 56, 56, 64)   256         conv2d_6[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_6 (ReLU)                  (None, 56, 56, 64)   0           batch_normalization_7[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_7 (Conv2D)               (None, 56, 56, 256)  16384       re_lu_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "add_1 (Add)                     (None, 56, 56, 256)  0           add[0][0]                        \n",
      "                                                                 conv2d_7[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_8 (BatchNor (None, 56, 56, 256)  1024        add_1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_7 (ReLU)                  (None, 56, 56, 256)  0           batch_normalization_8[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_8 (Conv2D)               (None, 56, 56, 64)   16384       re_lu_7[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, 56, 56, 64)   256         conv2d_8[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_8 (ReLU)                  (None, 56, 56, 64)   0           batch_normalization_9[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_9 (Conv2D)               (None, 56, 56, 64)   36864       re_lu_8[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_10 (BatchNo (None, 56, 56, 64)   256         conv2d_9[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_9 (ReLU)                  (None, 56, 56, 64)   0           batch_normalization_10[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_10 (Conv2D)              (None, 56, 56, 256)  16384       re_lu_9[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "add_2 (Add)                     (None, 56, 56, 256)  0           add_1[0][0]                      \n",
      "                                                                 conv2d_10[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_11 (BatchNo (None, 56, 56, 256)  1024        add_2[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_10 (ReLU)                 (None, 56, 56, 256)  0           batch_normalization_11[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_11 (Conv2D)              (None, 56, 56, 64)   16384       re_lu_10[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_12 (BatchNo (None, 56, 56, 64)   256         conv2d_11[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_11 (ReLU)                 (None, 56, 56, 64)   0           batch_normalization_12[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_12 (Conv2D)              (None, 56, 56, 64)   36864       re_lu_11[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_13 (BatchNo (None, 56, 56, 64)   256         conv2d_12[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_12 (ReLU)                 (None, 56, 56, 64)   0           batch_normalization_13[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_13 (Conv2D)              (None, 56, 56, 256)  16384       re_lu_12[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_3 (Add)                     (None, 56, 56, 256)  0           add_2[0][0]                      \n",
      "                                                                 conv2d_13[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_15 (BatchNo (None, 56, 56, 256)  1024        add_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_13 (ReLU)                 (None, 56, 56, 256)  0           batch_normalization_15[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_15 (Conv2D)              (None, 56, 56, 128)  32768       re_lu_13[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_16 (BatchNo (None, 56, 56, 128)  512         conv2d_15[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_14 (ReLU)                 (None, 56, 56, 128)  0           batch_normalization_16[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_16 (Conv2D)              (None, 28, 28, 128)  147456      re_lu_14[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_17 (BatchNo (None, 28, 28, 128)  512         conv2d_16[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_15 (ReLU)                 (None, 28, 28, 128)  0           batch_normalization_17[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_14 (BatchNo (None, 56, 56, 256)  1024        add_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_17 (Conv2D)              (None, 28, 28, 512)  65536       re_lu_15[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_14 (Conv2D)              (None, 28, 28, 512)  131072      batch_normalization_14[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_4 (Add)                     (None, 28, 28, 512)  0           conv2d_17[0][0]                  \n",
      "                                                                 conv2d_14[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_18 (BatchNo (None, 28, 28, 512)  2048        add_4[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_16 (ReLU)                 (None, 28, 28, 512)  0           batch_normalization_18[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_18 (Conv2D)              (None, 28, 28, 128)  65536       re_lu_16[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_19 (BatchNo (None, 28, 28, 128)  512         conv2d_18[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_17 (ReLU)                 (None, 28, 28, 128)  0           batch_normalization_19[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_19 (Conv2D)              (None, 28, 28, 128)  147456      re_lu_17[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_20 (BatchNo (None, 28, 28, 128)  512         conv2d_19[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_18 (ReLU)                 (None, 28, 28, 128)  0           batch_normalization_20[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_20 (Conv2D)              (None, 28, 28, 512)  65536       re_lu_18[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_5 (Add)                     (None, 28, 28, 512)  0           add_4[0][0]                      \n",
      "                                                                 conv2d_20[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_21 (BatchNo (None, 28, 28, 512)  2048        add_5[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_19 (ReLU)                 (None, 28, 28, 512)  0           batch_normalization_21[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_21 (Conv2D)              (None, 28, 28, 128)  65536       re_lu_19[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_22 (BatchNo (None, 28, 28, 128)  512         conv2d_21[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_20 (ReLU)                 (None, 28, 28, 128)  0           batch_normalization_22[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_22 (Conv2D)              (None, 28, 28, 128)  147456      re_lu_20[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_23 (BatchNo (None, 28, 28, 128)  512         conv2d_22[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_21 (ReLU)                 (None, 28, 28, 128)  0           batch_normalization_23[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_23 (Conv2D)              (None, 28, 28, 512)  65536       re_lu_21[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_6 (Add)                     (None, 28, 28, 512)  0           add_5[0][0]                      \n",
      "                                                                 conv2d_23[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_24 (BatchNo (None, 28, 28, 512)  2048        add_6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_22 (ReLU)                 (None, 28, 28, 512)  0           batch_normalization_24[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_24 (Conv2D)              (None, 28, 28, 128)  65536       re_lu_22[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_25 (BatchNo (None, 28, 28, 128)  512         conv2d_24[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_23 (ReLU)                 (None, 28, 28, 128)  0           batch_normalization_25[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_25 (Conv2D)              (None, 28, 28, 128)  147456      re_lu_23[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_26 (BatchNo (None, 28, 28, 128)  512         conv2d_25[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_24 (ReLU)                 (None, 28, 28, 128)  0           batch_normalization_26[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_26 (Conv2D)              (None, 28, 28, 512)  65536       re_lu_24[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_7 (Add)                     (None, 28, 28, 512)  0           add_6[0][0]                      \n",
      "                                                                 conv2d_26[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_27 (BatchNo (None, 28, 28, 512)  2048        add_7[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_25 (ReLU)                 (None, 28, 28, 512)  0           batch_normalization_27[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_27 (Conv2D)              (None, 28, 28, 128)  65536       re_lu_25[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_28 (BatchNo (None, 28, 28, 128)  512         conv2d_27[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_26 (ReLU)                 (None, 28, 28, 128)  0           batch_normalization_28[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_28 (Conv2D)              (None, 28, 28, 128)  147456      re_lu_26[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_29 (BatchNo (None, 28, 28, 128)  512         conv2d_28[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_27 (ReLU)                 (None, 28, 28, 128)  0           batch_normalization_29[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_29 (Conv2D)              (None, 28, 28, 512)  65536       re_lu_27[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_8 (Add)                     (None, 28, 28, 512)  0           add_7[0][0]                      \n",
      "                                                                 conv2d_29[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_31 (BatchNo (None, 28, 28, 512)  2048        add_8[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_28 (ReLU)                 (None, 28, 28, 512)  0           batch_normalization_31[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_31 (Conv2D)              (None, 28, 28, 256)  131072      re_lu_28[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_32 (BatchNo (None, 28, 28, 256)  1024        conv2d_31[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_29 (ReLU)                 (None, 28, 28, 256)  0           batch_normalization_32[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_32 (Conv2D)              (None, 14, 14, 256)  589824      re_lu_29[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_33 (BatchNo (None, 14, 14, 256)  1024        conv2d_32[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_30 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_33[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_30 (BatchNo (None, 28, 28, 512)  2048        add_8[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_33 (Conv2D)              (None, 14, 14, 1024) 262144      re_lu_30[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_30 (Conv2D)              (None, 14, 14, 1024) 524288      batch_normalization_30[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_9 (Add)                     (None, 14, 14, 1024) 0           conv2d_33[0][0]                  \n",
      "                                                                 conv2d_30[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_34 (BatchNo (None, 14, 14, 1024) 4096        add_9[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_31 (ReLU)                 (None, 14, 14, 1024) 0           batch_normalization_34[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_34 (Conv2D)              (None, 14, 14, 256)  262144      re_lu_31[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_35 (BatchNo (None, 14, 14, 256)  1024        conv2d_34[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_32 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_35[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_35 (Conv2D)              (None, 14, 14, 256)  589824      re_lu_32[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_36 (BatchNo (None, 14, 14, 256)  1024        conv2d_35[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_33 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_36[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_36 (Conv2D)              (None, 14, 14, 1024) 262144      re_lu_33[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_10 (Add)                    (None, 14, 14, 1024) 0           add_9[0][0]                      \n",
      "                                                                 conv2d_36[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_37 (BatchNo (None, 14, 14, 1024) 4096        add_10[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_34 (ReLU)                 (None, 14, 14, 1024) 0           batch_normalization_37[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_37 (Conv2D)              (None, 14, 14, 256)  262144      re_lu_34[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_38 (BatchNo (None, 14, 14, 256)  1024        conv2d_37[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_35 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_38[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_38 (Conv2D)              (None, 14, 14, 256)  589824      re_lu_35[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_39 (BatchNo (None, 14, 14, 256)  1024        conv2d_38[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_36 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_39[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_39 (Conv2D)              (None, 14, 14, 1024) 262144      re_lu_36[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_11 (Add)                    (None, 14, 14, 1024) 0           add_10[0][0]                     \n",
      "                                                                 conv2d_39[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_40 (BatchNo (None, 14, 14, 1024) 4096        add_11[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_37 (ReLU)                 (None, 14, 14, 1024) 0           batch_normalization_40[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_40 (Conv2D)              (None, 14, 14, 256)  262144      re_lu_37[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_41 (BatchNo (None, 14, 14, 256)  1024        conv2d_40[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_38 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_41[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_41 (Conv2D)              (None, 14, 14, 256)  589824      re_lu_38[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_42 (BatchNo (None, 14, 14, 256)  1024        conv2d_41[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_39 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_42[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_42 (Conv2D)              (None, 14, 14, 1024) 262144      re_lu_39[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_12 (Add)                    (None, 14, 14, 1024) 0           add_11[0][0]                     \n",
      "                                                                 conv2d_42[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_43 (BatchNo (None, 14, 14, 1024) 4096        add_12[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_40 (ReLU)                 (None, 14, 14, 1024) 0           batch_normalization_43[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_43 (Conv2D)              (None, 14, 14, 256)  262144      re_lu_40[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_44 (BatchNo (None, 14, 14, 256)  1024        conv2d_43[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_41 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_44[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_44 (Conv2D)              (None, 14, 14, 256)  589824      re_lu_41[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_45 (BatchNo (None, 14, 14, 256)  1024        conv2d_44[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_42 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_45[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_45 (Conv2D)              (None, 14, 14, 1024) 262144      re_lu_42[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_13 (Add)                    (None, 14, 14, 1024) 0           add_12[0][0]                     \n",
      "                                                                 conv2d_45[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_46 (BatchNo (None, 14, 14, 1024) 4096        add_13[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_43 (ReLU)                 (None, 14, 14, 1024) 0           batch_normalization_46[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_46 (Conv2D)              (None, 14, 14, 256)  262144      re_lu_43[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_47 (BatchNo (None, 14, 14, 256)  1024        conv2d_46[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_44 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_47[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_47 (Conv2D)              (None, 14, 14, 256)  589824      re_lu_44[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_48 (BatchNo (None, 14, 14, 256)  1024        conv2d_47[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_45 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_48[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_48 (Conv2D)              (None, 14, 14, 1024) 262144      re_lu_45[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_14 (Add)                    (None, 14, 14, 1024) 0           add_13[0][0]                     \n",
      "                                                                 conv2d_48[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_49 (BatchNo (None, 14, 14, 1024) 4096        add_14[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_46 (ReLU)                 (None, 14, 14, 1024) 0           batch_normalization_49[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_49 (Conv2D)              (None, 14, 14, 256)  262144      re_lu_46[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_50 (BatchNo (None, 14, 14, 256)  1024        conv2d_49[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_47 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_50[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_50 (Conv2D)              (None, 14, 14, 256)  589824      re_lu_47[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_51 (BatchNo (None, 14, 14, 256)  1024        conv2d_50[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_48 (ReLU)                 (None, 14, 14, 256)  0           batch_normalization_51[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_51 (Conv2D)              (None, 14, 14, 1024) 262144      re_lu_48[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_15 (Add)                    (None, 14, 14, 1024) 0           add_14[0][0]                     \n",
      "                                                                 conv2d_51[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_53 (BatchNo (None, 14, 14, 1024) 4096        add_15[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_49 (ReLU)                 (None, 14, 14, 1024) 0           batch_normalization_53[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_53 (Conv2D)              (None, 14, 14, 512)  524288      re_lu_49[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_54 (BatchNo (None, 14, 14, 512)  2048        conv2d_53[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_50 (ReLU)                 (None, 14, 14, 512)  0           batch_normalization_54[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_54 (Conv2D)              (None, 7, 7, 512)    2359296     re_lu_50[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_55 (BatchNo (None, 7, 7, 512)    2048        conv2d_54[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_51 (ReLU)                 (None, 7, 7, 512)    0           batch_normalization_55[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_52 (BatchNo (None, 14, 14, 1024) 4096        add_15[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_55 (Conv2D)              (None, 7, 7, 2048)   1048576     re_lu_51[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_52 (Conv2D)              (None, 7, 7, 2048)   2097152     batch_normalization_52[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_16 (Add)                    (None, 7, 7, 2048)   0           conv2d_55[0][0]                  \n",
      "                                                                 conv2d_52[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_56 (BatchNo (None, 7, 7, 2048)   8192        add_16[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_52 (ReLU)                 (None, 7, 7, 2048)   0           batch_normalization_56[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_56 (Conv2D)              (None, 7, 7, 512)    1048576     re_lu_52[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_57 (BatchNo (None, 7, 7, 512)    2048        conv2d_56[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_53 (ReLU)                 (None, 7, 7, 512)    0           batch_normalization_57[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_57 (Conv2D)              (None, 7, 7, 512)    2359296     re_lu_53[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_58 (BatchNo (None, 7, 7, 512)    2048        conv2d_57[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_54 (ReLU)                 (None, 7, 7, 512)    0           batch_normalization_58[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_58 (Conv2D)              (None, 7, 7, 2048)   1048576     re_lu_54[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_17 (Add)                    (None, 7, 7, 2048)   0           add_16[0][0]                     \n",
      "                                                                 conv2d_58[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_59 (BatchNo (None, 7, 7, 2048)   8192        add_17[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_55 (ReLU)                 (None, 7, 7, 2048)   0           batch_normalization_59[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_59 (Conv2D)              (None, 7, 7, 512)    1048576     re_lu_55[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_60 (BatchNo (None, 7, 7, 512)    2048        conv2d_59[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_56 (ReLU)                 (None, 7, 7, 512)    0           batch_normalization_60[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_60 (Conv2D)              (None, 7, 7, 512)    2359296     re_lu_56[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_61 (BatchNo (None, 7, 7, 512)    2048        conv2d_60[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_57 (ReLU)                 (None, 7, 7, 512)    0           batch_normalization_61[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_61 (Conv2D)              (None, 7, 7, 2048)   1048576     re_lu_57[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_18 (Add)                    (None, 7, 7, 2048)   0           add_17[0][0]                     \n",
      "                                                                 conv2d_61[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_62 (BatchNo (None, 7, 7, 2048)   8192        add_18[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_58 (ReLU)                 (None, 7, 7, 2048)   0           batch_normalization_62[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_62 (Conv2D)              (None, 7, 7, 512)    1048576     re_lu_58[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_63 (BatchNo (None, 7, 7, 512)    2048        conv2d_62[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_59 (ReLU)                 (None, 7, 7, 512)    0           batch_normalization_63[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_63 (Conv2D)              (None, 7, 7, 512)    2359296     re_lu_59[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_64 (BatchNo (None, 7, 7, 512)    2048        conv2d_63[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "re_lu_60 (ReLU)                 (None, 7, 7, 512)    0           batch_normalization_64[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_64 (Conv2D)              (None, 7, 7, 2048)   1048576     re_lu_60[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "add_19 (Add)                    (None, 7, 7, 2048)   0           add_18[0][0]                     \n",
      "                                                                 conv2d_64[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling2d (Globa (None, 2048)         0           add_19[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, 1)            2049        global_average_pooling2d[0][0]   \n",
      "==================================================================================================\n",
      "Total params: 29,482,817\n",
      "Trainable params: 29,426,113\n",
      "Non-trainable params: 56,704\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.layers import Dense, GlobalAveragePooling2D\n",
    "from tensorflow.keras.regularizers import l2\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "\n",
    "# TODO: Subclass the Model\n",
    "\n",
    "from resnet_v2_c import ResNetV2\n",
    "unet = ResNetV2(50, input_shape=(224, 224, 1), reg=l2(0.001))\n",
    "unet.compile(loss='mse', optimizer=Adam(lr=0.001), metrics=['mse'])\n",
    "unet.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-Train the Model\n",
    "\n",
    "### Warmup (Numerical Stability)\n",
    "\n",
    "Do warmup training on a small number of epochs, starting at a very tiny learning rate and incremently raising it to the (presumed) initial learning rate for full training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Warmup (for numerical stability)\n",
      "Train on 832 samples\n",
      "\n",
      "Epoch 00001: LearningRateScheduler reducing learning rate to 9.999999974752427e-07.\n",
      "Epoch 1/5\n",
      "832/832 [==============================] - 24s 29ms/sample - loss: 65.6005 - mean_squared_error: 0.9545\n",
      "\n",
      "Epoch 00002: LearningRateScheduler reducing learning rate to 0.00019980000000000003.\n",
      "Epoch 2/5\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 66.5685 - mean_squared_error: 2.0151\n",
      "*** Loss is diverging, Reducing Warmnup Rate\n",
      "\n",
      "Epoch 00003: LearningRateScheduler reducing learning rate to 3.9960000000000004e-05.\n",
      "Epoch 3/5\n",
      "832/832 [==============================] - 13s 16ms/sample - loss: 64.6442 - mean_squared_error: 0.2100\n",
      "\n",
      "Epoch 00004: LearningRateScheduler reducing learning rate to 5.994000000000001e-05.\n",
      "Epoch 4/5\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 64.6089 - mean_squared_error: 0.2377\n",
      "\n",
      "Epoch 00005: LearningRateScheduler reducing learning rate to 7.992000000000001e-05.\n",
      "Epoch 5/5\n",
      "832/832 [==============================] - 12s 15ms/sample - loss: 64.4138 - mean_squared_error: 0.1417\n"
     ]
    }
   ],
   "source": [
    "# Hyperparameters\n",
    "LR=0.001 # The initial learning rate for full training\n",
    "BS=32     # Batch Size\n",
    "\n",
    "# Do warmup training for numerical stability of the weights\n",
    "unet.warmup(x_train, y_train, e_lr=LR, batch_size=BS, loss='mse', metrics=['mse'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Tuning\n",
    "\n",
    "Do a grid search on the best initial learning rate, using very few epochs/steps per trial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Hyperparameter Grid Search\n",
      "*** Learning Rate 0.0001\n",
      "Epoch 1/3\n",
      "250/250 [==============================] - 136s 546ms/step - loss: 64.1297 - mean_squared_error: 0.6333\n",
      "Epoch 2/3\n",
      "250/250 [==============================] - 121s 482ms/step - loss: 61.2217 - mean_squared_error: 0.0792\n",
      "Epoch 3/3\n",
      "250/250 [==============================] - 121s 484ms/step - loss: 58.2023 - mean_squared_error: 0.1320\n",
      "44/44 [==============================] - 2s 51ms/sample - loss: 56.6058 - mean_squared_error: 0.1896\n",
      "*** Learning Rate 0.001\n",
      "Epoch 1/3\n",
      "250/250 [==============================] - 136s 544ms/step - loss: 75.6497 - mean_squared_error: 12.4355\n",
      "Epoch 2/3\n",
      "250/250 [==============================] - 120s 480ms/step - loss: 57.9824 - mean_squared_error: 0.1654\n",
      "Epoch 3/3\n",
      "250/250 [==============================] - 120s 481ms/step - loss: 51.2399 - mean_squared_error: 0.0532\n",
      "44/44 [==============================] - 2s 38ms/sample - loss: 47.8580 - mean_squared_error: 0.0572\n",
      "*** Learning Rate 0.01\n",
      "Epoch 1/3\n",
      "250/250 [==============================] - 137s 550ms/step - loss: 1185.7691 - mean_squared_error: 1039.6824\n",
      "Epoch 2/3\n",
      "250/250 [==============================] - 120s 482ms/step - loss: 141.0880 - mean_squared_error: 0.3896\n",
      "Epoch 3/3\n",
      "250/250 [==============================] - 121s 482ms/step - loss: 128.6347 - mean_squared_error: 0.2314\n",
      "44/44 [==============================] - 2s 42ms/sample - loss: 122.6195 - mean_squared_error: 0.1407\n",
      "*** Selected best learning rate: 0.001\n",
      "*** Selected best batch size: 32\n"
     ]
    }
   ],
   "source": [
    "# Do grid search on three magnitudes of learning rate\n",
    "LR, _ = unet.grid_search(x_train, y_train, x_test, y_test, epochs=3, steps=250, lr_range=[0.0001, 0.001, 0.01], batch_range=[32], loss='mse', metrics=['mse'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "\n",
    "Train the model using the learning rate picked by automatic hyperparameter tuning with cosine decay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Full Training\n",
      "Train on 832 samples, validate on 44 samples\n",
      "\n",
      "Epoch 00001: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 1/45\n",
      "832/832 [==============================] - 23s 27ms/sample - loss: 127.4243 - mean_squared_error: 63.2198 - val_loss: 24628962.9091 - val_mean_squared_error: 24628898.0000\n",
      "\n",
      "Epoch 00002: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 2/45\n",
      "832/832 [==============================] - 13s 16ms/sample - loss: 64.9409 - mean_squared_error: 0.6070 - val_loss: 12628.4045 - val_mean_squared_error: 12564.1807\n",
      "\n",
      "Epoch 00003: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 3/45\n",
      "832/832 [==============================] - 13s 16ms/sample - loss: 64.1993 - mean_squared_error: 0.1298 - val_loss: 78.2230 - val_mean_squared_error: 14.3354\n",
      "\n",
      "Epoch 00004: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 4/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 63.7514 - mean_squared_error: 0.0540 - val_loss: 70.8389 - val_mean_squared_error: 7.3563\n",
      "\n",
      "Epoch 00005: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 5/45\n",
      "832/832 [==============================] - 12s 15ms/sample - loss: 63.3067 - mean_squared_error: 0.0389 - val_loss: 69.1663 - val_mean_squared_error: 6.1383\n",
      "\n",
      "Epoch 00006: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 6/45\n",
      "832/832 [==============================] - 12s 15ms/sample - loss: 62.8371 - mean_squared_error: 0.0442 - val_loss: 66.2989 - val_mean_squared_error: 3.7665\n",
      "\n",
      "Epoch 00007: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 7/45\n",
      "832/832 [==============================] - 12s 15ms/sample - loss: 62.3048 - mean_squared_error: 0.0248 - val_loss: 64.1953 - val_mean_squared_error: 2.1935\n",
      "\n",
      "Epoch 00008: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 8/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 61.7745 - mean_squared_error: 0.0399 - val_loss: 62.3744 - val_mean_squared_error: 0.9334\n",
      "\n",
      "Epoch 00009: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 9/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 61.1988 - mean_squared_error: 0.0376 - val_loss: 61.9021 - val_mean_squared_error: 1.0475\n",
      "\n",
      "Epoch 00010: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 10/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 60.6032 - mean_squared_error: 0.0395 - val_loss: 60.8526 - val_mean_squared_error: 0.6067\n",
      "\n",
      "Epoch 00011: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 11/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 59.9850 - mean_squared_error: 0.0394 - val_loss: 60.3813 - val_mean_squared_error: 0.7632\n",
      "\n",
      "Epoch 00012: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 12/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 59.3984 - mean_squared_error: 0.0886 - val_loss: 59.0518 - val_mean_squared_error: 0.0773\n",
      "\n",
      "Epoch 00013: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 13/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 58.7828 - mean_squared_error: 0.1233 - val_loss: 58.3619 - val_mean_squared_error: 0.0445\n",
      "\n",
      "Epoch 00014: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 14/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 58.0879 - mean_squared_error: 0.0910 - val_loss: 57.6771 - val_mean_squared_error: 0.0282\n",
      "\n",
      "Epoch 00015: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 15/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 57.3534 - mean_squared_error: 0.0301 - val_loss: 57.0031 - val_mean_squared_error: 0.0330\n",
      "\n",
      "Epoch 00016: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 16/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 56.7132 - mean_squared_error: 0.0727 - val_loss: 56.3259 - val_mean_squared_error: 0.0426\n",
      "\n",
      "Epoch 00017: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 17/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 56.0232 - mean_squared_error: 0.0725 - val_loss: 55.8577 - val_mean_squared_error: 0.2673\n",
      "\n",
      "Epoch 00018: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 18/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 55.3298 - mean_squared_error: 0.0745 - val_loss: 54.9826 - val_mean_squared_error: 0.0901\n",
      "\n",
      "Epoch 00019: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 19/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 54.6101 - mean_squared_error: 0.0550 - val_loss: 54.2303 - val_mean_squared_error: 0.0400\n",
      "\n",
      "Epoch 00020: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 20/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 53.8895 - mean_squared_error: 0.0376 - val_loss: 53.6321 - val_mean_squared_error: 0.1463\n",
      "\n",
      "Epoch 00021: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 21/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 53.1961 - mean_squared_error: 0.0498 - val_loss: 52.8221 - val_mean_squared_error: 0.0426\n",
      "\n",
      "Epoch 00022: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 22/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 52.4891 - mean_squared_error: 0.0495 - val_loss: 52.1032 - val_mean_squared_error: 0.0307\n",
      "\n",
      "Epoch 00023: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 23/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 51.8132 - mean_squared_error: 0.0804 - val_loss: 51.4019 - val_mean_squared_error: 0.0359\n",
      "\n",
      "Epoch 00024: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 24/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 51.0972 - mean_squared_error: 0.0705 - val_loss: 50.7544 - val_mean_squared_error: 0.0938\n",
      "\n",
      "Epoch 00025: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 25/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 50.4540 - mean_squared_error: 0.1317 - val_loss: 50.4276 - val_mean_squared_error: 0.4700\n",
      "\n",
      "Epoch 00026: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 26/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 49.7306 - mean_squared_error: 0.1100 - val_loss: 49.2994 - val_mean_squared_error: 0.0422\n",
      "\n",
      "Epoch 00027: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 27/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 48.9796 - mean_squared_error: 0.0584 - val_loss: 48.6414 - val_mean_squared_error: 0.0826\n",
      "\n",
      "Epoch 00028: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 28/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 48.2843 - mean_squared_error: 0.0599 - val_loss: 47.9412 - val_mean_squared_error: 0.0775\n",
      "\n",
      "Epoch 00029: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 29/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 47.5815 - mean_squared_error: 0.0506 - val_loss: 47.2465 - val_mean_squared_error: 0.0744\n",
      "\n",
      "Epoch 00030: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 30/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 46.9246 - mean_squared_error: 0.0833 - val_loss: 46.5357 - val_mean_squared_error: 0.0506\n",
      "\n",
      "Epoch 00031: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 31/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 46.2297 - mean_squared_error: 0.0729 - val_loss: 46.0845 - val_mean_squared_error: 0.2815\n",
      "\n",
      "Epoch 00032: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 32/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 45.5566 - mean_squared_error: 0.0797 - val_loss: 45.1588 - val_mean_squared_error: 0.0332\n",
      "\n",
      "Epoch 00033: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 33/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 44.8349 - mean_squared_error: 0.0332 - val_loss: 44.4883 - val_mean_squared_error: 0.0355\n",
      "\n",
      "Epoch 00034: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 34/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 44.1629 - mean_squared_error: 0.0315 - val_loss: 43.9204 - val_mean_squared_error: 0.1351\n",
      "\n",
      "Epoch 00035: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 35/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 43.5334 - mean_squared_error: 0.0668 - val_loss: 43.1522 - val_mean_squared_error: 0.0285\n",
      "\n",
      "Epoch 00036: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 36/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 42.8640 - mean_squared_error: 0.0559 - val_loss: 42.5197 - val_mean_squared_error: 0.0516\n",
      "\n",
      "Epoch 00037: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 37/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 42.2252 - mean_squared_error: 0.0697 - val_loss: 41.9075 - val_mean_squared_error: 0.0886\n",
      "\n",
      "Epoch 00038: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 38/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 41.5610 - mean_squared_error: 0.0516 - val_loss: 41.2100 - val_mean_squared_error: 0.0341\n",
      "\n",
      "Epoch 00039: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 39/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 40.9022 - mean_squared_error: 0.0331 - val_loss: 40.5744 - val_mean_squared_error: 0.0356\n",
      "\n",
      "Epoch 00040: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 40/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 40.2951 - mean_squared_error: 0.0601 - val_loss: 39.9814 - val_mean_squared_error: 0.0732\n",
      "\n",
      "Epoch 00041: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 41/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 39.7130 - mean_squared_error: 0.1049 - val_loss: 39.6453 - val_mean_squared_error: 0.3600\n",
      "\n",
      "Epoch 00042: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 42/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 39.0179 - mean_squared_error: 0.0298 - val_loss: 38.7430 - val_mean_squared_error: 0.0749\n",
      "\n",
      "Epoch 00043: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 43/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 38.4015 - mean_squared_error: 0.0276 - val_loss: 38.0987 - val_mean_squared_error: 0.0414\n",
      "\n",
      "Epoch 00044: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 44/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 37.7957 - mean_squared_error: 0.0293 - val_loss: 37.4873 - val_mean_squared_error: 0.0339\n",
      "\n",
      "Epoch 00045: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
      "Epoch 45/45\n",
      "832/832 [==============================] - 13s 15ms/sample - loss: 37.2190 - mean_squared_error: 0.0532 - val_loss: 36.8820 - val_mean_squared_error: 0.0254\n"
     ]
    }
   ],
   "source": [
    "EPOCHS=45\n",
    "\n",
    "# add the test data back into the training data. During training, 5% will be split off as validation data\n",
    "x_train = np.concatenate((x_train, x_test))\n",
    "y_train = np.concatenate((y_train, y_test))\n",
    "#unet.training(x_train, y_train, epochs=EPOCHS, batch_size=BS, lr=LR, decay=('cosine', 0), loss='mse', metrics=['mse'], split=0.05)\n",
    "unet.training(x_train, y_train, epochs=EPOCHS, batch_size=BS, lr=LR, loss='mse', metrics=['mse'], split=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predictions\n",
    "\n",
    "### Today\n",
    "\n",
    "Let's first predict using yesterday's data for daily new cases for today and compare to the actual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# US\n",
    "image = preprocess_input('covid-images/US-47-1.0.jpg')\n",
    "batch = np.asarray([image])\n",
    "batch = np.expand_dims(batch, -1)\n",
    "print(\"US\", unet.model.predict(batch), 1.0)\n",
    "\n",
    "# Canada\n",
    "image = preprocess_input('covid-images/CA-47-0.7426367461430575.jpg')\n",
    "batch = np.asarray([image])\n",
    "batch = np.expand_dims(batch, -1)\n",
    "print(\"CA\", unet.model.predict(batch), 0.74)\n",
    "\n",
    "# Great Britian\n",
    "image = preprocess_input('covid-images/UK-47-1.0.jpg')\n",
    "batch = np.asarray([image])\n",
    "batch = np.expand_dims(batch, -1)\n",
    "print(\"GB\", unet.model.predict(batch), 1.0)\n",
    "\n",
    "# Germany\n",
    "image = preprocess_input('covid-images/DE-47-0.9780743565300286.jpg')\n",
    "batch = np.asarray([image])\n",
    "batch = np.expand_dims(batch, -1)\n",
    "print(\"DE\", unet.model.predict(batch), 0.97)\n",
    "\n",
    "# Spain\n",
    "image = preprocess_input('covid-images/ES-47-0.8370201691607027.jpg')\n",
    "batch = np.asarray([image])\n",
    "batch = np.expand_dims(batch, -1)\n",
    "print(\"ES\", unet.model.predict(batch), 0.84)\n",
    "\n",
    "# Iran\n",
    "image = preprocess_input('covid-images/IR-47-0.9375392341494037.jpg')\n",
    "batch = np.asarray([image])\n",
    "batch = np.expand_dims(batch, -1)\n",
    "print(\"IR\", unet.model.predict(batch), 0.94)\n",
    "\n",
    "# Italy\n",
    "image = preprocess_input('covid-images/IT-47-0.729296934573738.jpg')\n",
    "batch = np.asarray([image])\n",
    "batch = np.expand_dims(batch, -1)\n",
    "print(\"IT\", unet.model.predict(batch), 0.73)\n",
    "\n",
    "# France\n",
    "image = preprocess_input('covid-images/FR-47-0.6414621272103458.jpg')\n",
    "batch = np.asarray([image])\n",
    "batch = np.expand_dims(batch, -1)\n",
    "print(\"FR\", unet.model.predict(batch), 0.64)\n",
    "\n",
    "# Austrialia\n",
    "image = preprocess_input('covid-images/AU-47-0.4402618657937807.jpg')\n",
    "batch = np.asarray([image])\n",
    "batch = np.expand_dims(batch, -1)\n",
    "print(\"AU\", unet.model.predict(batch), 0.44)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tomorrow\n",
    "\n",
    "Let's now predict tomorrow's daily new cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "US 23379\n",
      "CA 980\n",
      "GB 2482\n",
      "AU 203\n",
      "IT 2994\n",
      "ES 7642\n",
      "DE 5107\n",
      "FR 2082\n",
      "IR 2902\n"
     ]
    }
   ],
   "source": [
    "GB=\"GB\"  # ECDC code\n",
    "\n",
    "countries = ['US', 'CA', GB, 'AU', 'IT', 'ES', 'DE', 'FR', 'IR']\n",
    "for country in countries:\n",
    "    image = preprocess_input('covid-images/' + country + '- last.jpg')\n",
    "    batch = np.asarray([image])\n",
    "    batch = np.expand_dims(batch, -1)\n",
    "    for data in all_data:\n",
    "        if data[0] == country:\n",
    "            max = data[2]\n",
    "            break\n",
    "    print(country, int(unet.model.predict(batch) * max))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
